Nonlinear Knowledge in Spline Models

Support vector machines (SVMs) are a useful tool for learning classifiers or function approximations from data [8, 15, 16, 2, 17, 3, 9]. One important interpretation of SVMs is as an optimization problem in a reproducing-kernel Hilbert space (RKHS) [20, 21]. Recently, prior knowledge in the form of inequalities which must be satisfied over sets of the input space have been added to SVMs for both classification [5, 4] and approximation [10, 12]. In particular, [12] treated knowledge consisting of nonlinear inequalities imposed over arbitrary sets. However, these previous approaches did not consider the RKHS interpretation of the function approximation problem. Instead, the kernel was treated simply as a basis function which was used to generate a nonlinear function approximation. In the present work, we will show that the Kimeldorf and Wahba representer theorem [6] allows us to use the optimization techniques used by Mangasarian and Wild [12] to add nonlinear prior knowledge to spline models using inequality constrained splines [19, 20]. Section 2 reviews the prior knowledge formulation introduced in [12] and shows how it may be converted into an infinite system of linear inequality constraints. These constraints are enforced via an inequality constrained spline model in Section 3, where it is also shown that the resulting problem remains an optimization problem in an RKHS. Computational results are reported in Section 4. Section 5 concludes the paper and discusses avenues of future research. We describe our notation now. All vectors will be column vectors unless transposed to a row vector by a prime ′. The scalar (inner) product of two vectors x and y in the n-dimensional real space R will be denoted by x′y. Note that this notation will be used only to denote the inner product in R, we will use a different notation when we wish to denote the inner product in an arbitrary RKHS. A vector of zeros in a real space of arbitrary dimension will be denoted by 0. The notation A ∈ Rm×n will signify a real m× n matrix. For such a matrix, A′ will denote the transpose of A, Ai will denote the i-th row of A and A·j the j-th column of A. The identity matrix of arbitrary dimension will be denoted by I.